Data stream processing meets the Advanced Metering Infrastructure: possibilities, challenges and applications

Distribution of electricity is changing.Energy production is increasingly distributed, weather dependent and located in the distribution network, close to consumers.Energy consumption is increasing throughout society and the electrification of transportation is driving distribution networks closer to the limits.Operating the networks closer to their limits also increases the risk for faults.Continuous monitoring of the distribution network closest to the customers is needed in order to mitigate this risk.The Advanced Metering Infrastructure introduced smart meters throughout the distribution network.Data stream processing is a computing paradigm that offers low latency results from analysis on large volumes of the data.This thesis investigates the possibilities and challenges for continuous monitoring that are created when the Advanced Metering Infrastructure and data stream processing meet.The challenges that are addressed in the thesis are efficient processing of unordered (also called out-of-order) data and efficient usage of the computational resources present in the Advanced Metering Infrastructure.Contributions towards more efficient processing of out-of-order data are made with eChIDNA and TinTiN. Both are systems that utilize knowledge about smart meter data to directly produce results where possible and storing only data that is relevant for late data in order to produce updated results when such late data arrives. eChIDNA is integrated in the streaming query itself, while TinTiN is a streaming middleware that can be applied to streaming queries in order to make them resilient against out-of-order data.Eventual determinism is defined in order to formally investigate the deterministic properties of output produced by such systems.Contributions towards efficient usage of the computational resources of the Advanced Metering Infrastructure are made with the application LoCoVolt.LoCoVolt implements a monitoring algorithm that can run on equipment that is localized in the communication infrastructure of the Advanced Metering Infrastructure and can take advantage of the overlap between the communication and distribution networks.All contributions are evaluated on hardware that is available in current AMI systems, using large scale data obtained from a real production AMI

TinTiN: Travelling in time (if necessary) to deal with out-of-order data in streaming aggregation

Cyber-Physical Systems (CPS) rely on data stream processing for high-throughput, low-latency analysis with correctness and accuracy guarantees (building on deterministic execution) for monitoring, safety or security applications.The trade-offs in processing performance and results\u27 accuracy are nonetheless application-dependent. While some applications need strict deterministic execution, others can value fast (but possibly approximated) answers.Despite the existing literature on how to relax and trade strict determinism for efficiency or deadlines, we lack a formal characterization of levels of determinism, needed by industries to assess whether or not such trade-offs are acceptable.To bridge the gap, we introduce the notion of D-bounded eventual determinism, where D is the maximum out-of-order delay of the input data.We design and implement TinTiN, a streaming middleware that can be used in combination with user-defined streaming applications, to provably enforce D-bounded eventual determinism.We evaluate TinTiN with a real-world streaming application for Advanced Metering Infrastructure (AMI) monitoring, showing it provides an order of magnitude improvement in processing performance, while minimizing delays in output generation, compared to a state-of-the-art strictly deterministic solution that waits for time proportional to D, for each input tuple, before generating output that depends on it

eChIDNA: Continuous Data Validation in Advanced Metering Infrastructures

New laws and regulations increase the demands for a more data-intense metering infrastructure towards more adaptive electricity networks (aka smart grids). The automatic measuring, often involving wireless communication, introduces errors both in software and during data transmission. These demands, as well as the large data volumes that need to be validated, present new challenges to utilities. First, measurement errors cannot be allowed to propagate to the data stored by utilities. Second, manual fixing of errors after storing is not a feasible option with increasing data volumes and decreasing lead times for new services and analysis. Third, validation is not only to be applied to current readings but also to past readings when new types of errors are discovered. This paper addresses these issues by proposing a hybrid system, eChIDNA, utilizing both the store-then-process and the data streaming processing paradigms, enabling for high throughput, low latency distributed and parallel analysis. Validation rules are built upon this paradigm and then implemented on the state of the art Apache Storm Stream Processing Engine to assess performance. Furthermore, patterns of common errors are matched, triggering alerts as a first step towards automatic correction of errors. The system is evaluated with production data from hundreds of thousands of smart meters. The results show a performance in the thousands messages per second realm, showing that stream processing can be used to validate large volumes of meter data online with low processing latency, identifying common errors as they appear. The results from the pattern matching are cross-validated with system experts and show that pattern matching is a viable way to minimize time required from human operators

LoCoVolt: Distributed Detection of Broken Meters in Smart Grids through Stream Processing

Smart Grids and Advanced Metering Infrastructures are rapidly replacing traditional energy grids.The cumulative computational power of their IT devices, which can be leveraged to continuously monitor the state of the grid, is nonetheless vastly underused.This paper provides evidence of the potential of streaming analysis run at smart grid devices.We propose a structural component, which we name \name{} (Local Comparison of Voltages), that is able to detect in a distributed fashion malfunctioning smart meters, which report erroneous information about the power quality. This is achieved by comparing the voltage readings of meters that, because of their proximity in the network, are expected to report readings following similar trends. Having this information can allow utilities to react promptly and thus increase timeliness, quality and safety of their services to society and, implicitly, their business value.As we show, based on our implementation on Apache Flink and the evaluation conducted with resource-constrained hardware (i.e., with capacity similar to that of hardware in smart grids) and data from a real-world network, the streaming paradigm can deliver efficient and effective monitoring tools and thus achieve the desired goals with almost no additional computational cost

Occupational exposure to gases/fumes and mineral dust affect DNA methylation levels of genes regulating expression

Many workers are daily exposed to occupational agents like gases/fumes, mineral dust or biological dust, which could induce adverse health effects. Epigenetic mechanisms, such as DNA methylation, have been suggested to play a role. We therefore aimed to identify differentially methylated regions (DMRs) upon occupational exposures in never-smokers and investigated if these DMRs associated with gene expression levels. To determine the effects of occupational exposures independent of smoking, 903 never-smokers of the LifeLines cohort study were included. We performed three genome-wide methylation analyses (Illumina 450 K), one per occupational exposure being gases/fumes, mineral dust and biological dust, using robust linear regression adjusted for appropriate confounders. DMRs were identified using comb-p in Python. Results were validated in the Rotterdam Study (233 never-smokers) and methylation-expression associations were assessed using Biobank-based Integrative Omics Study data (n = 2802). Of the total 21 significant DMRs, 14 DMRs were associated with gases/fumes and 7 with mineral dust. Three of these DMRs were associated with both exposures (RPLP1 and LINC02169 (2x)) and 11 DMRs were located within transcript start sites of gene expression regulating genes. We replicated two DMRs with gases/fumes (VTRNA2-1 and GNAS) and one with mineral dust (CCDC144NL). In addition, nine gases/fumes DMRs and six mineral dust DMRs significantly associated with gene expression levels. Our data suggest that occupational exposures may induce differential methylation of gene expression regulating genes and thereby may induce adverse health effects. Given the millions of workers that are exposed daily to occupational exposures, further studies on this epigenetic mechanism and health outcomes are warranted

CT angiography and CT perfusion improve prediction of infarct volume in patients with anterior circulation stroke

Introduction: We investigated whether baseline CT angiography (CTA) and CT perfusion (CTP) in acute ischemic stroke could improve prediction of infarct presence and infarct volume on follow-up imaging. Methods: We analyzed 906 patients with suspected anterior circulation stroke from the prospective multicenter Dutch acute stroke study (DUST). All patients underwent baseline non-contrast CT, CTA, and CTP and follow-up non-contrast CT/MRI after 3 days. Multivariable regression models were developed including patient characteristics and non-contrast CT, and subsequently, CTA and CTP measures were added. The increase in area under the curve (AUC) and R2 was assessed to determine the additional value of CTA and CTP. Results: At follow-up, 612 patients (67.5 %) had a detectable infarct on CT/MRI; median infarct volume was 14.8 mL (interquartile range (IQR) 2.8–69.6). Regarding infarct presence, the AUC of 0.82 (95 % confidence interval (CI) 0.79–0.85) for patient characteristics and non-contrast CT was improved with addition of CTA measures (AUC 0.85 (95 % CI 0.82–0.87); p < 0.001) and was even higher after addition of CTP measures (AUC 0.89 (95 % CI 0.87–0.91); p < 0.001) and combined CTA/CTP measures (AUC 0.89 (95 % CI 0.87–0.91); p < 0.001). For infarct volume, adding combined CTA/CTP measures (R2 = 0.58) was superior to patient characteristics and non-contrast CT alone (R2 = 0.44) and to addition of CTA alone (R2 = 0.55) or CTP alone (R2 = 0.54; all p < 0.001). Conclusion: In the acute stage, CTA and CTP have additional value over patient characteristics and non-contrast CT for predicting infarct presence and infarct volume on follow-up imaging. These findings could be applied for patient selection in future trials on ischemic stroke treatment

Prediction of outcome in patients with suspected acute ischaemic stroke with CT perfusion and CT angiography: The Dutch acute stroke trial (DUST) study protocol

Background: Prediction of clinical outcome in the acute stage of ischaemic stroke can be difficult when based on patient characteristics, clinical findings and on non-contrast CT. CT perfusion and CT angiography may provide additional prognostic information and guide treatment in the early stage. We present the study protocol of the Dutch acute Stroke Trial (DUST). The DUST aims to assess the prognostic value of CT perfusion and CT angiography in predicting stroke outcome, in addition to patient characteristics and non-contrast CT. For this purpose, individualised prediction models for clinical outcome after stroke based on the best predictors from patient characteristics and CT imaging will be developed and validated.Methods/design: The DUST is a prospective multi-centre cohort study in 1500 patients with suspected acute ischaemic stroke. All patients undergo non-contrast CT, CT perfusion and CT angiography within 9 hours after onset of the neurological deficits, and, if possible, follow-up imaging after 3 days. The primary outcome is a dichotomised score on the modified Rankin Scale, assessed at 90 days. A score of 0-2 represents good outcome, and a score of 3-6 represents poor outcome. Three logistic regression models will be developed, including patient characteristics and non-contrast CT (model A), with addition of CT angiography (model B), and CT perfusion parameters (model C). Model derivation will be performed in 60% of the study population, and model validation in the remaining 40% of the patients. Additional prognostic value of the models will be determined with the area under the curve (AUC) from the receiver operating characteristic (ROC) curve, calibration plots, assessment of goodness-of-fit, and likelihood ratio tests.Discussion: This study will provide insight in the added prognosti

Controlling bias and inflation in epigenome- and transcriptome-wide association studies using the empirical null distribution

We show that epigenome- and transcriptome-wide association studies (EWAS and TWAS) are prone to significant inflation and bias of test statistics, an unrecognized phenomenon introducing spurious findings if left unaddressed. Neither GWAS-based methodology nor state-of-the-art confounder adjustment methods completely remove bias and inflation. We propose a Bayesian method to control bias and inflation in EWAS and TWAS based on estimation of the empirical null distribution. Using simulations and real data, we demonstrate that our method maximizes power while properly controlling the false positive rate. We illustrate the utility of our method in large-scale EWAS and TWAS meta-analyses of age and smoking